initial SVE intrinsics by davidtwco · Pull Request #2071 · rust-lang/stdarch

davidtwco · 2026-04-05T12:48:30Z

This is a very large patch but the vast majority of it is generated code, the rest should be relatively straightforward to review.

This passes tests for me locally with the latest nightly after rust-lang/rust#153286 has landed.

I've bolded the commits that do the most interesting things - the rest are largely updating the support that has been partially upstreamed in the past to work with the current implementation in rustc.

a6a76cb re-adds the "missing run-time test" check in stdarch-verify for Arm - it was accidentally removed in 713444d.
099249b removes some special-casing in the generator for files being generated from sve.spec.yml and sve2.spec.yml and instead adds a key that can be set in those files to achieve the same behaviour.
8d352c0 removes SvUndef expressions from the generator, these were used in svundef intrinsics and some others. svundef now zeroes and so doesn't need a special generator expression, and the other uses call svundef.
b97d6fb changes uses of simd_reinterpret in the generator to transmute_unchecked - simd_reinterpret was expected to be added by rustc_scalable_vector(N) rust#143924 but was changed during reviews.
0934280 changes the behaviour of auto-llvm-sign-conversion in the generator for return types to match what it does for arguments. auto-llvm-sign-conversion isn't used by any existing intrinsic specification files.
4832b15 adds a static_assert_range helper that will be used in later commits.
cddc125 defines the scalable vector types and adds the modules that the generated intrinsics will eventually live in. This defines some additional non-public scalable vector types necessary to match the signatures of the LLVM intrinsics, and some traits for conversions between those and the public scalable vector types.
d754502 changes the generator to generate calls to SveInto::sve_into (a trait introduced in the previous commit). This trait is necessary because target_feature annotations are necessary on the trait method and that requires the method be unsafe, which is incompatible with the signature of Into::into.
bc28f52 adds the intrinsic definitions - there's a lot of definitions and it likely isn't too practical to look over all of them. We later check that everything that should be being generated is being generated using stdarch-verify. Load/store tests will also be generated from these definitions.
b8feb57 runs the generator after the above commit to generate the Rust code for all the intrinsic definitions.
ebe8860 updates the intrinsics_data/arm_intrinsics.json file to add all the SVE intrinsics so that stdarch-verify can check all of the intrinsics are present.
5bdaf54 updates stdarch-verify with the necessary support and special-cases for SVE intrinsics.
7203b17 updates the intrinsic-test tool so that it can still parse arm_intrinsics.json and skips the SVE intrinsics.

The intrinsic test tool needs a lot of changes to work with SVE and this patch is getting big as it is, so we're going to do this as a follow-up (agreed in advance with @Amanieu).

This patch is based on #1509 and I've tried to preserve the co-authorship of everyone involved in that patch.

r? @Amanieu

This was accidentally removed in 713444d.

Instead of generating load/store tests based on the input filename - which no longer works given the expected input file structure of `stdarch-gen-arm` - add a simple global context option that SVE specs can set.

The `SvUndef` expression is no longer necessary as a `core::intrinsics::scalable::sve_undef` intrinsic has been introduced to produce an undefined SVE vector, used by `svundef*` intrinsics. Other intrinsics that used `SvUndef` now use the `svundef*` intrinsics.

`simd_reinterpret` was expected to be used when it was added as `transmute_unchecked` requires `Sized`, but scalable vectors are now `Sized` so `transmute_unchecked` can be used and `simd_reinterpret` was not added in rust-lang/rust#143924.

Matching the current behaviour for arguments, `auto_llvm_sign_conversion` should only be required for `as_unsigned` conversions, not `into` conversions.

davidtwco · 2026-04-05T12:55:55Z

Looks like this'll need another rustc patch to skip the "scalable vectors aren't supported on this architecture" error for doc builds

davidtwco · 2026-04-05T13:44:17Z

rust-lang/rust#154850 will fix this

…, r=JonathanBrouwer ast_validation: scalable vectors okay for rustdoc Scalable vector types in `core_arch` are cfg'd for aarch64 and for rustdoc, which can successfully document these types given any `--target` (`core_arch` CI uses `i686-unknown-linux-gnu`) - this shouldn't trigger the "scalable vectors not supported on arch" error. This fixes the CI failure in rust-lang/stdarch#2071.

Rollup merge of #154850 - davidtwco:scalable-vectors-rustdoc, r=JonathanBrouwer ast_validation: scalable vectors okay for rustdoc Scalable vector types in `core_arch` are cfg'd for aarch64 and for rustdoc, which can successfully document these types given any `--target` (`core_arch` CI uses `i686-unknown-linux-gnu`) - this shouldn't trigger the "scalable vectors not supported on arch" error. This fixes the CI failure in rust-lang/stdarch#2071.

davidtwco · 2026-04-07T12:09:04Z

rust-lang/rust#154950 will fix the current CI failure

crates/core_arch/src/aarch64/sve/mod.rs

library: no `cfg(target_arch)` on scalable intrinsics These intrinsics don't technically need to be limited to a specific architecture, they'll probably only make sense to use on AArch64, but this just makes it harder to use them in stdarch where it is appropriate (such as on `arm64ec`): requiring a rustc patch to land and be on nightly before stdarch work can proceed. So let's just not `cfg` them at all, they're perma-unstable anyway. Fixes CI failure in rust-lang/stdarch#2071

Rollup merge of #154950 - davidtwco:scalable-no-cfg, r=RalfJung library: no `cfg(target_arch)` on scalable intrinsics These intrinsics don't technically need to be limited to a specific architecture, they'll probably only make sense to use on AArch64, but this just makes it harder to use them in stdarch where it is appropriate (such as on `arm64ec`): requiring a rustc patch to land and be on nightly before stdarch work can proceed. So let's just not `cfg` them at all, they're perma-unstable anyway. Fixes CI failure in rust-lang/stdarch#2071

library: no `cfg(target_arch)` on scalable intrinsics These intrinsics don't technically need to be limited to a specific architecture, they'll probably only make sense to use on AArch64, but this just makes it harder to use them in stdarch where it is appropriate (such as on `arm64ec`): requiring a rustc patch to land and be on nightly before stdarch work can proceed. So let's just not `cfg` them at all, they're perma-unstable anyway. Fixes CI failure in rust-lang/stdarch#2071

This is a convenience macro used by the generated SVE intrinsics. Co-authored-by: Jamie Cunliffe <Jamie.Cunliffe@arm.com> Co-authored-by: Luca Vizzarro <Luca.Vizzarro@arm.com> Co-authored-by: Adam Gemmell <adam.gemmell@arm.com> Co-authored-by: Jacob Bramley <jacob.bramley@arm.com>

adamgemmell

Just to add that we did pretty thorough manual reviews of each intrinsic at the time, and very few should be untested once the intrinsic-test changes go in (svundef comes to mind)

View changes since this review

adamgemmell · 2026-04-08T10:20:45Z

crates/stdarch-gen-arm/src/intrinsic.rs

-                    }
-                })
-        };
+        let return_type_conversion = self


It's probably the case that this signed/unsigned conversion isn't needed at all. Doesn't really cause any harm though

adamgemmell · 2026-04-08T10:36:12Z

crates/stdarch-verify/src/lib.rs

                );
            }

+            // Newer intrinsics don't have `rustc_legacy_const_generics` - assume they belong at


We could add this attribute for SVE intrinsics - see rust-lang/rust#149654. Some have pointed out that legacy generics is prone to bugs, but:

It aids porting from C in a small way

RfL really wants const args. I'm not sure if they necessarily want it for SIMD intrinsics though.

davidtwco · 2026-04-08T11:15:10Z

rust-lang/rust#154984 will fix this CI failure, eventually I'll just have issues on the stdarch side :)

Add the SVE types (without any of the generated intrinsics) and empty modules where the generated intrinsics will be. Enables the `adt_const_params` crate feature that the generated intrinsics will use. Co-authored-by: Jamie Cunliffe <Jamie.Cunliffe@arm.com> Co-authored-by: Luca Vizzarro <Luca.Vizzarro@arm.com> Co-authored-by: Adam Gemmell <adam.gemmell@arm.com> Co-authored-by: Jacob Bramley <jacob.bramley@arm.com>

`Into::into` can't be used here because the implementations can't have the required target feature, so `SveInto` needs to be introduced and written by the generator

`core::ptr::from_exposed_addr` was renamed to `core::ptr::with_exposed_provenance` and so this link needs updated.

Thousands of lines of SVE intrinsic definitions.. Co-authored-by: Jamie Cunliffe <Jamie.Cunliffe@arm.com> Co-authored-by: Luca Vizzarro <Luca.Vizzarro@arm.com> Co-authored-by: Adam Gemmell <adam.gemmell@arm.com> Co-authored-by: Jacob Bramley <jacob.bramley@arm.com>

Following from previous commit, this commit only contains generated code from the SVE intrinsic specifications Co-authored-by: Jamie Cunliffe <Jamie.Cunliffe@arm.com> Co-authored-by: Luca Vizzarro <Luca.Vizzarro@arm.com> Co-authored-by: Adam Gemmell <adam.gemmell@arm.com> Co-authored-by: Jacob Bramley <jacob.bramley@arm.com>

Co-authored-by: Adam Gemmell <Adam.Gemmell@arm.com> Co-authored-by: Jamie Cunliffe <Jamie.Cunliffe@arm.com> Co-authored-by: Jacob Bramley <jacob.bramley@arm.com> Co-authored-by: Luca Vizzarro <Luca.Vizzarro@arm.com>

arm64ec doesn't support SVE.

With SVE intrinsics in the `arm_intrinsics.json`, the parsing needs to be updated to know to expect any new fields.

Amanieu · 2026-04-05T23:02:29Z

crates/core_arch/src/macros.rs


+#[allow(unused_macros)]
+macro_rules! static_assert_range {
+    ($imm:ident, $min:literal, $max:literal) => {


View changes since the review

($imm:ident, $min:literal..=$max:literal) would make it clearer that this is an inclusive range.

Amanieu · 2026-04-05T23:53:29Z

crates/core_arch/src/aarch64/sve/mod.rs

+
+macro_rules! impl_sve_type {
+    ($(($v:vis, $elem_type:ty, $name:ident, $elt:literal))*) => ($(
+        #[derive(Clone, Copy)]


View changes since the review

Have you checked whether this actually compiles when built as part of core? I expect this to fail because these public types have no documentation on them, which would trigger a denied lint.

I copied over the lints that are enabled in core and checked that they pass - added some documentation comments to do that

crates/core_arch/src/aarch64/sve/mod.rs

Amanieu · 2026-04-05T23:55:13Z

crates/core_arch/src/lib.rs

@@ -1,4 +1,7 @@
 #![doc = include_str!("core_arch_docs.md")]
+// FIXME(scalable-vectors): Required for `adt_const_params` Used for unit-only enums in SVE
+// intrinsics.


View changes since the review

These could also be defined as plain integer consts instead. In the end I don't mind either way but it may delay stabilization if we are blocked on adt_const_params.

I've changed this to min_adt_const_params and removed incomplete_features - I've checked with @BoxyUwU and this should be pretty close to stabilisation so I feel comfortable relying on it for now

davidtwco · 2026-04-08T15:47:32Z

rust-lang/rust#154984 will fix this CI failure, eventually I'll just have issues on the stdarch side :)

I've closed this and changed the patch to only add these intrinsics on target_arch = "aarch64" for now.

davidtwco added 5 commits April 5, 2026 12:24

stdarch-verify: re-add runtime test check

a6a76cb

This was accidentally removed in 713444d.

gen-arm: add generate_load_store_tests

099249b

Instead of generating load/store tests based on the input filename - which no longer works given the expected input file structure of `stdarch-gen-arm` - add a simple global context option that SVE specs can set.

gen-arm: remove SvUndef

8d352c0

The `SvUndef` expression is no longer necessary as a `core::intrinsics::scalable::sve_undef` intrinsic has been introduced to produce an undefined SVE vector, used by `svundef*` intrinsics. Other intrinsics that used `SvUndef` now use the `svundef*` intrinsics.

gen-arm: auto-llvm-sign-conversion not for into

0934280

Matching the current behaviour for arguments, `auto_llvm_sign_conversion` should only be required for `as_unsigned` conversions, not `into` conversions.

davidtwco added the A-aarch64 label Apr 5, 2026

rustbot assigned Amanieu Apr 5, 2026

davidtwco mentioned this pull request Apr 5, 2026

ast_validation: scalable vectors okay for rustdoc rust-lang/rust#154850

Merged

davidtwco force-pushed the sve-intrinsics branch from 7203b17 to d473c9a Compare April 7, 2026 11:11

davidtwco mentioned this pull request Apr 7, 2026

library: no cfg(target_arch) on scalable intrinsics rust-lang/rust#154950

Merged

bjorn3 reviewed Apr 7, 2026

View reviewed changes

crates/core_arch/src/aarch64/sve/mod.rs Show resolved Hide resolved

adamgemmell reviewed Apr 8, 2026

View reviewed changes

davidtwco mentioned this pull request Apr 8, 2026

target: support scalable vectors on arm64ec rust-lang/rust#154984

Closed

davidtwco force-pushed the sve-intrinsics branch from d473c9a to 6a6098b Compare April 8, 2026 11:52

davidtwco and others added 8 commits April 8, 2026 12:15

gen-arm: use sve_into instead of into

d75f15f

`Into::into` can't be used here because the implementations can't have the required target feature, so `SveInto` needs to be introduced and written by the generator

gen-arm: correct renamed from_exposed_addr link

ac41fa2

`core::ptr::from_exposed_addr` was renamed to `core::ptr::with_exposed_provenance` and so this link needs updated.

intrinsics_data: add sve intrinsics

bf5e289

Co-authored-by: Adam Gemmell <Adam.Gemmell@arm.com> Co-authored-by: Jamie Cunliffe <Jamie.Cunliffe@arm.com> Co-authored-by: Jacob Bramley <jacob.bramley@arm.com> Co-authored-by: Luca Vizzarro <Luca.Vizzarro@arm.com>

stdarch-verify: support sve

98027ea

Co-authored-by: Adam Gemmell <Adam.Gemmell@arm.com> Co-authored-by: Jamie Cunliffe <Jamie.Cunliffe@arm.com> Co-authored-by: Jacob Bramley <jacob.bramley@arm.com> Co-authored-by: Luca Vizzarro <Luca.Vizzarro@arm.com>

core_arch: no SVE on arm64ec

8a59be6

arm64ec doesn't support SVE.

intrinsic-test: update parsing for SVE intrinsics

9401448

With SVE intrinsics in the `arm_intrinsics.json`, the parsing needs to be updated to know to expect any new fields.

Amanieu reviewed Apr 8, 2026

View reviewed changes

davidtwco force-pushed the sve-intrinsics branch from 6a6098b to 9401448 Compare April 8, 2026 15:45

Conversation

davidtwco commented Apr 5, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidtwco commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidtwco commented Apr 5, 2026

Uh oh!

davidtwco commented Apr 7, 2026

Uh oh!

Uh oh!

adamgemmell left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgemmell Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

adamgemmell Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

davidtwco commented Apr 8, 2026

Uh oh!

Amanieu Apr 5, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidtwco Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Amanieu Apr 5, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidtwco Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Amanieu Apr 5, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidtwco Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

davidtwco commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

davidtwco commented Apr 5, 2026 •

edited by rustbot

Loading

davidtwco commented Apr 5, 2026 •

edited

Loading

adamgemmell left a comment •

edited by rustbot

Loading

Amanieu Apr 5, 2026 •

edited by rustbot

Loading

Amanieu Apr 5, 2026 •

edited by rustbot

Loading

Amanieu Apr 5, 2026 •

edited by rustbot

Loading